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Program Description^ 

The Incredible Years is composed of training programs for chiidren, 
parents, and teachers. The chiid program is designed for children 
(ages 0-12) with challenging behaviors and focuses on building social 
and emotional skills. Lessons can be delivered to children referred for 
difficult behavior or to an entire classroom as a preventative measure. 

The program consists of 20- to 30-minute lessons two to three times a 
week; these lessons are reinforced by small-group activities, practicing 
skills throughout the day, and communicating with parents. Lessons 
cover recognizing and understanding feelings, getting along with 
friends, anger management, problem solving, and behavior at school. 

Parent training programs focus on positive discipline, promoting learn- 
ing and development, and involvement in children’s life at school. 

Research^ 

One study of The Incredible Years that falls within the scope of 
the Children Classified as Having an Emotional Disturbance review 
protocol meets What Works Clearinghouse (WWC) evidence stan- 
dards. This one study included 51 four- to eight-year-old children with 
oppositional defiant disorder who attended school in Washington 

state. Based on this one study, the WWC considers the extent of evidence for The Incredible Years on children 
classified as having an emotional disturbance (or children at risk for classification) to be small for the external 
behavior and social outcomes domains. 
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Effectiveness 

The Incredible Years was found to have potentially positive effects on external behavior and potentially positive 
effects on social outcomes for children classified as having an emotional disturbance. 



Table 1. Summary of findings^ 







Improvement index (percentile points) 








Outcome domain 


Rating of effectiveness 


Average 


Range 


Number of 
Studies 


Number of 
Students 


Extent of 
evidence 


External behavior 


Potentially positive effects 


+20 


+18 to +22 


1 


51 


Small 


Social outcomes 


Potentially positive effects 


+18 


na 


1 


51 


Small 



na = not applicable 
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Program Information 

Background 

The Incredible Years was developed by Carolyn Webster-Stratton, a professor and director of the Parenting Clinic 
at the University of Washington, and is distributed by The Incredible Years. Address: 141 1 8th Avenue West, 

Seattle, WA98119. Email: incredibleyears@incredibleyears.com. Web: http://www.incredibleyears.com/. 

Telephone: (888) 506-3562 or (206) 285-7565. Fax: (888) 506-3562. 

Program details 

The Incredible Years is designed for children (ages 0-12) with challenging behaviors and their teachers and parents. 
The program has been used with children diagnosed with conduct problems (e.g., having high rates of aggression, 
defiance, and oppositional and impulsive behaviors) and attention deficit disorder. It also has been used with culturally 
diverse groups, including Hispanic/Latino, Asian-American, African-American, and new immigrant families. 

The Incredible Years is composed of training programs for children, parents, and teachers. The child program 
focuses on building social and emotional skills during preschool through early elementary school and can be 
delivered to children referred for difficult behavior or to an entire classroom as a preventive measure. In a class- 
room setting, the teacher presents 20- to 30-minute lessons two to three times a week during circle time, which are 
reinforced by small-group activities; encouragement of skills practice throughout the day; and communication with 
parents, including home activities in which they can participate. Lessons cover recognizing and understanding feelings, 
getting along with friends, anger management, problem solving, and behavior at school. In a clinical setting, referred 
students complete activities during 18 to 20 two-hour weekly small-group meetings. Ideally, parents of children in 
the clinical program are involved in the parent training module. Parent training programs focus on parenting skills, 
including positive discipline, promoting learning and development, and ways to be involved in their children’s lives 
at school. The Incredible Years also includes two programs for teachers. The first addresses classroom management 
as a means to improve student behavior and learning. The second is a training program for teachers who will 
deliver the program in their classrooms. 

Cost* 

Total per-child costs range from $1 ,164 to $3,003, depending on which components are used. 
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Research Summary 



Seventy-seven studies reviewed by the WWC investigated the 
effects of The Incredible Years on children classified as having an 
emotional disturbance (or chiidren at risk for classification). One 
study (Webster-Stratton, Reid, & Hammond, 2004) is a randomized 
controlied trial that meets WWC evidence standards. The remaining 
76 studies do not meet either WWC eiigibility screens or evidence 
standards. (See references beginning on page 5 for citations for 
all 77 studies.) 



Table 2. Scope of reviewed research 



Grade 


PK, K, 1, 2 


Delivery method 


Small group, Individual 


Program type 


Curriculum, Supplement 


Studies reviewed 


77 


Meets WWC standards 


1 study 


Meets WWC standards 
with reservations 


0 studies 



Summary of studies meeting WWC evidence standards without reservations 

Webster-Stratton et ai. (2004) randomiy assigned 159 families to one of six conditions: 

• Parent training alone (PT) 

• Child training aione (CT) 

• Parent training pius teacher training (PT + TT) 

• Child training plus teacher training (CT + TT) 

• Parent and child training combined with teacher training (PT + CT + TT) 

• Wait-iist comparison group 

Twenty-five students received the fuli version of The Incredible Years (PT + CT + TT), and 26 students were in the 
comparison group.® Participants were recruited from famiiies requesting treatment at the University of Washington 
Parenting Clinic. The primary referral problem was child misconduct (e.g., noncompiiance, aggression, oppositionai 
behaviors) that had been occurring for at least six months. Famiiies entered the study in three cohorts (50 to 55 
families per cohort) in the fail of 1995, 1996, and 1997. Random assignment was conducted by lottery after all fami- 
lies in the cohort had completed baseline assessments. The sample included students in preschool, kindergarten, first 
grade, and second grade. The study reported outcomes after approximately six months (mid-November to April) of 
program implementation.® 

Summary of studies meeting WWC evidence standards with reservations 

No studies of The Incredible Years met WWC evidence standards with reservations. 
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Effectiveness Summary 

The WWC review of interventions for Chiidren Classified as Having an Emotionai Disturbance addresses student 
outcomes in seven domains: external behavior, emotional/internai behavior, social outcomes, reading achievement/ 
iiteracy, math achievement, school attendance, and other academic performance. The one study that infiuences 
the findings in this report covers two domains: external behavior and sociai outcomes.^ The findings beiow present 
the authors’ estimates and WWC-calcuiated estimates of the size and statistical significance of the effects of The 
Incredible Years on children classified as having an emotionai disturbance. For a more detailed description of the 
rating of effectiveness and extent of evidence criteria, see Appendix E. 

Summary of effectiveness for the external behavior domain 

Webster-Stratton et ai. (2004) found, and the WWC confirmed, three statisticaiiy significant differences between the 
treatment and comparison group on Chiid Conduct Problems (CCP). These measures are (1) CCP at Home: Mother 
Report, (2) CCP at Home: Father Report, and (3) CCP at School. Thus, for the external behavior domain, this one 
study with a strong design showed statistically significant, potentially positive effects.® This results in a rating of 
potentiaily positive effects, with a small extent of evidence. 



Table 3. Rating of effectiveness and extent of evidence for the externai behavior domain 



Rating of effectiveness 


Criteria met 


Potentially positive effects 

Evidence of a positive effect with 
no overriding contrary evidence. 


The review of The Incredible Years had one study showing a statistically significant positive effect, no studies showing 
a statistically significant or substantively important negative effect, and no studies showing indeterminate effects. 


Extent of evidence 


Criteria met 


Small 


The review of The Incredible Years had a) one study, AND b) an unknown number of schools, AND c) 51 students. 



Summary of effectiveness for the social outcomes domain 

Webster-Stratton et ai. (2004) found, and the WWC confirmed, one statisticaiiy significant difference between the 
treatment and comparison groups on Child Social Competence (CSC) with Peers. Thus, for the social outcomes 
domain, this one study with a strong design showed a statistically significant, potentially positive effect. This results 
in a rating of potentially positive effects, with a small extent of evidence. 



Table 4. Rating of effectiveness and extent of evidence for the sociai outcomes domain 



Rating of effectiveness 


Criteria met 


Potentially positive effects 

Evidence of a positive effect with 
no overriding contrary evidence. 


The review of The Incredible Years had one study showing a statistically significant positive effect, no studies showing 
a statistically significant or substantively important negative effect, and no studies showing indeterminate effects. 


Extent of evidence 


Criteria met 


Small 


The review of The Incredible /ears had a) one study, AND b) an unknown number of schools, AND c) 51 students. 
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Appendix A: Research details for Webster-Stratton et al., 2004 

Webster-Stratton, C., Reid, M. J., & Hammond, M. (2004). Treating chiidren with eariy-onset conduct 
probiems: Intervention outcomes for parent, chiid, and teacher training. Journal of Clinical Child 
and Adolescent Psychology, 33(1), 105-124. 



Table Al. Summary of findings Meets WWC evidence standards 







Study findings 


Outcome domain 


Sample size 


Average improvement index 

(percentile points) 


Statistically significant 


External behavior 


51 students 


+20 


Yes 


Social outcomes 


51 students 


+18 


Yes 



Setting The teacher, parent, and child trainings were primarily instituted in the University of Washing- 
ton Parenting Clinic. The children assigned to treatment conditions came to the clinic’s “Dino- 
saur School.” Participants then practiced their skills at home and in the classroom. 



Study sample a sample of 1 59 families was randomly assigned to one of six conditions: parent training alone 
(PT; n = 31); child training alone (CT; n = 30); parent training plus teacher training (PT + TT; n = 
24); child training plus teacher training (CT + TT; n = 23); parent and child training combined 
with teacher training (PT + CT -r TT; n = 25); or a wait-list comparison group (n = 26). The final 
sample sizes for each of the six conditions differ by outcome measure. Participants were 
recruited from families requesting treatment at the University of Washington Parenting Clinic. 
Families were self-referred or referred by professionals in the community (20% by teachers 
and 38% by physicians). The primary referral problem was child misconduct (e.g., noncom- 
pliance, aggression, oppositional behaviors) that had been occurring for at least six months. 
Families entered the study in three cohorts (50 to 55 families per cohort) in the fall of 1995, 
1996, and 1997. Random assignment was conducted by lottery after all families in the cohort 
had completed baseline assessments. The student sample was predominantly European 
American (79%), 90% were boys, and the mean age was 71 months. The sample consisted of 
students in preschool, kindergarten, first grade, and second grade. 

IntcrVGntion The children assigned to CT, CT -i- TT, and CT + PT + TT conditions came to the clinic’s Dinosaur 
group School for 2 hours each week for 18 to 19 weeks (lasting approximately six months) and met with 
two therapists. The Dinosaur School program specifically addressed interpersonal difficulties that 
are problematic for young children with oppositional defiant disorder (ODD). Weekly letters were 
sent to teachers and parents explaining the key concepts and the rationale for the targeted skill 
(e.g., sharing, teamwork, friendly talk, listening, compliance to requests, feeling talk, and problem 
solving). Teachers and parents were asked to reinforce the targeted social skills whenever they 
noticed the child using them in the home or school, and children were given weekly homework 
assignments to complete with their parents. The parents assigned to PT, PT + TT, and PT + CT + 
TT conditions met at the clinic each week for a 2-hour session. Over the course of 22 to 24 weeks, 
they watched 17 videotape programs on parenting and interpersonal skills designed to reduce 
parents’ coercive interactions and strengthen positive interactions and relationships with their 
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Comparison 

group 



Outcomes and 
measurement 



Support for 
implementation 



children. Teachers in the PT + TT, CT + TT, and PT + CT + TT conditions came to the clinic for 4 
full days (32 hours) of group training sequenced throughout the school year, to correspond roughly 
with the beginning, first quarter, second quarter, and end of the PT and CT treatments. 

The families assigned to the comparison condition received no treatment from the Parenting 
Clinic and had no contact with therapists during the 8- to 9-month waiting period. These fami- 
lies were offered the parent training program after the outcomes from the first year of the study 
had been measured. 

This study included measures of Child Conduct Problems (CCP) at Home, Child Conduct 
Problems (CCP) at School, and Child Social Competence (CSC) with Peers. For a more 
detailed description of these outcome measures, see Appendix B. 

The teacher curriculum targeted teachers’ use of effective classroom management strate- 
gies for handling misbehavior, promoting positive relationships with difficult students, and 
strengthening social skills In all school settings. Workshop topics Included promoting social 
skills through praise and encouragement, proactive teaching, using incentives to motivate 
children, strategies to decrease disruptive behavior, and collaborative approaches for working 
with parents. Teachers also learned to prevent peer rejection by helping the aggressive child 
learn appropriate problem-solving strategies and by helping his or her peers respond appro- 
priately to aggression. Teachers were trained to have age-appropriate expectations and to 
be sensitive to individual developmental differences and biological deficits in children, and to 
understand the relevance of these differences for enhanced teaching efforts that are positive, 
accepting, and consistent. To ensure the integrity of the treatment, therapists co-led their first 
parent or child group with a supervisor, completed a weekly checklist of standards, and were 
monitored weekly. All child and parent sessions were videotaped for feedback and analyses, 
and the supervisor randomly selected videotapes for fidelity checks. Analyses Indicated that 
all required videotape vignettes were shown and that all required homework was assigned. 
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Appendix B: Outcome measures for each domain 



External behavior 


Child Conduct Problems (CCP) at Home: 
Mother Report 


This composite score inciudes the Eyberg Chiid Behavior Inventory (ECBi) Total Intensity score and four inde- 
pendent observations ot child behavior in the home using the Dyadic Parent-Child Interaction Coding System- 
Revised (DPICS-R) and the Coder Impressions Inventory (Cll). The ECBI is a 36-item behavioral inventory of child 
conduct problem behavior for children ages 2 to 16 and was completed by the mother; the ECBI Total Intensity 
score was used as an indicator of the frequency with which problem behaviors occur. The DPICS-R, originally 
developed by Robinson and Eyberg (1981) and revised by Webster-Stratton (1989), is an observational measure 
for recording behaviors of children in their home; this composite score included the DPICS-R total child deviance 
(sum of whine, cry, physical negative, smart talk, yell, and destructive) plus noncompliance (child does not begin 
to comply to parent command within 5 seconds) variables and a one-item rating of child affect. This composite 
score also included two single-item variables from the Cll (percentage of time child acted inappropriately and 
total overall poor conduct) (as cited in Webster-Stratton et al., 2004). 


Child Conduct Problems (CCP) at Home: 
Father Report 


This composite score was identical to the CCP at Home: Mother Report with one exception — the ECBI Total 
Intensity score was completed by the father for the CCP at Home: Father Report (as cited in as cited in Webster- 
Stratton et al., 2004). 


Child Conduct Problems (CCP) at School 


This composite score includes two teacher report variables: Teacher Assessment of School Behavior (TASB; 
aggressive behavior scale) and the Teacher Rating scales of Perceived Competence Scale for Young Children 
(PCSC; behavior conduct score). The composite score also includes two summary scores from independent 
observations in the classroom using Multi-Option Observation System for Experimental Studies (MOOSES) 
(frequency of child negative behaviors with teachers and peers) and observation ratings of poor authority 
acceptance from the Social Health Profile (SHP). The composite score also includes the Dyadic Peer Interaction 
Scale (DPIS), a measure of total inappropriate behavior with peers (e.g., dyad was loud, physically active, 
impulsive, reckless), from independent observations of the child interacting with a peer in a laboratory (as cited 
in Webster-Stratton et al., 2004). 


Social outcomes 


Child Social Competence (CSC) 
with Peers 


This composite score includes two teacher-report variables: the social acceptance scores from the Teacher 
Assessment of School Behavior (TASB) and from the Perceived Competence Scale for Young Children (PCSC). 
This composite score also includes one classroom observation variable, the social contact score on the Social 
Health Profile (SHP), and one laboratory observation variable, the positive communication score on the Dyadic 
Peer Interaction Scale (DPIS) (as cited in Webster-Stratton et al., 2004). 
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Appendix C.1: Findings inciuded in the rating for the externai behavior domain 



Mean 

(standard deviation) WWC calculations 

Study Sample Intervention Comparison Mean Effect Improvement 



Outcome measure 


sample 


size 


group 


group 


difference 


size 


index 


p-value 


Webster-Stratton et al., 


2004= 
















Child Conduct Problems 
(CCP) at Home: Mother 
Report 


Ages 4-8 
(PT -F TT 
-fCT) 


51 


36.99 

(10.46) 


47.28 

(9.79) 


10.29 


0.57 


22 


<0.05 


Child Conduct Problems 
(CCP) at School 


Ages 4-8 
(PT -F TT 
-fCT) 


51 


29.58 

(14.23) 


39.55 

(18.69) 


9.97 


0.46 


18 


<0.05 


Domain average for external behavior across one study 








0.52 


20 


Statistically 

significant 



Table Notes: This appendix reports findings considered for the effectiveness rating and the average improvement indices for the externai behavior domain. Positive resuits for mean 
difference, effect size, and improvement index favor the intervention group; negative resuits favor the comparison group. For the CCP at Home: Mother Report and CCP at Schooi, signs 
were reversed on the mean difference, effect size, and improvement index to demonstrate that the treatment group was favored when negative differences were reported. The effect 
size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that 
can be expected if that student is given the intervention. The improvement index is an aiternate presentation of the effect size, refiecting the change in an average student's percentiie 
rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simpie average rounded to two decimai piaces; the average improve- 
ment index is caicuiated from the average effect size. The statisticai significance of the study's domain average was determined by the WWC; the study is characterized as having a 
statisticaiiy significant positive effect because univariate statisticai tests are reported for each outcome measure, the effect for at ieast one measure within the domain is positive and 
statisticaiiy significant, and no effects are negative and statisticaiiy significant. PT = Parent Training; TT = Teacher Training; CT = Chiid Training. 

® Findings from contrasts between the comparison group and other treatment conditions (PT, CT, PT -i- TT, and CT + TT) from Webster-Stratton et ai. (2004) are not inciuded in these 
ratings but are reported in Appendix D.1 . The means reported here were adjusted for pretest scores by the study authors. The p-vaiue ranges presented here were reported in the 
originai study (Reid et ai., 2003). A correction for muitipie comparisons was needed but did not affect significance ieveis. The originai study reported findings on three externai 
behavior outcomes; the father-reported outcomes are not presented in this appendix due to high attrition. The WWC muitipie comparison correction accounts for the fact that three 
measures were used in the articie to measure externai behavior. 



Appendix C.2: Findings inciuded in the rating for the sociai outcomes domain 







Mean 

(standard deviation) 


WWC calculations 




Study 

Outcome measure sample 


Sample 

size 


Intervention 

group 


Comparison 

group 


Mean 

difference 


Effect 

size 


Improvement 

index 


p-value 


Webster-Stratton et al., 2004= 


Child Social Competence Ages 4-8 

(CSC) with Peers (PT -f TT 

-fCT) 


51 


53.15 

(12.77) 


44.13 

(15.41) 


9.02 


0.46 


18 


<0.05 


Domain average for social outcomes across one study 








0.46 


18 


Statistically 

significant 



Table Notes: This appendix reports findings considered for the effectiveness rating and the average improvement indices for the sociai outcomes domain. Positive resuits for mean 
difference, effect size, and improvement index favor the intervention group; negative resuits favor the comparison group. The effect size is a standardized measure of the effect of 
an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can be expected if that student is given the 
intervention. The improvement index is an aiternate presentation of the effect size, refiecting the change in an average student’s percentiie rank that can be expected if the student 
is given the intervention. PT = Parent Training; TT = Teacher Training; CT = Chiid Training. 

Findings from contrasts between the comparison group and other treatment conditions (PT, CT, PT -i- TT, and CT + TT) from Webster-Stratton et ai. (2004) are not inciuded in these 
ratings but are reported in Appendix D.1 . The means reported here were adjusted for pretest scores by the study authors. The p-vaiue range presented here was reported in the 
originai study (Reid et ai., 2003). No corrections for ciustering or muitipie comparisons were needed. 
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Appendix D.1: Partial implementation findings for the external behavior domain 









Mean 

(standard deviation) 


WWC calculations 




Outcome measure 


Study 

sample 


Sample 

size 


Intervention 

group 


Comparison 

group 


Mean 

difference 


Effect 

size 


Improvement 

index 


p-value 


Webster-Stratton et al., 


2004= 
















Child Conduct Problems 
(CCP) at Home: Mother 
Report 


Ages 4-8 
(PT) 


57 


35.96 

(11.47) 


47.28 

(9.79) 


11.32 


0.67 


25 


<0.05 


Child Conduct Problems 
(CCP) at Home: Mother 
Report 


Ages 4-8 
(PT -H TT) 


50 


39.99 

(12.22) 


47.28 

(9.79) 


7.29 


0.41 


16 


<0.05 


Child Conduct Problems 
(CCP) at Home: Mother 
Report 


Ages 4-8 
(CT) 


55 


40.19 

(12.19) 


47.28 

(9.79) 


7.09 


0.41 


16 


<0.05 


Child Conduct Problems 
(CCP) at Home: Mother 
Report 


Ages 4-8 
(CT -H TT) 


49 


37.25 

(9.83) 


47.28 

(9.79) 


10.03 


0.55 


21 


<0.05 


Child Conduct Problems 
(CCP) at School 


Ages 4-8 
(PT) 


56 


32.50 

(14.42) 


39.55 

(18.69) 


7.05 


0.35 


14 


<0.05 


Child Conduct Problems 
(CCP) at School 


Ages 4-8 
(PT -H TT) 


50 


31.06 

(16.32) 


39.55 

(18.69) 


8.49 


0.41 


16 


<0.05 


Child Conduct Problems 
(CCP) at School 


Ages 4-8 
(CT) 


56 


32.23 

(14.73) 


39.55 

(18.69) 


7.32 


0.41 


16 


<0.05 


Child Conduct Problems 
(CCP) at School 


Ages 4-8 
(CT -H TT) 


49 


32.49 

(15.44) 


39.55 

(18.69) 


7.06 


0.41 


16 


<0.05 



Table Notes: This appendix presents comparisons between chiidren who received partiai versions of The Incredible Years (PT, CT, PT + TT, and CT + TT) and chiidren in the com- 
parison group on measures that faii in the externai behavior domain. These are anciiiary comparisons for the purposes of this review. Comparisons between chiidren who received 
the fuii version of The Incredible Years and chiidren in the comparison group were used for rating purposes and are presented in Appendix C.1 . Positive resuits for mean difference, 
effect size, and improvement index favor the intervention group; negative resuits favor the comparison group. For the CCP at Home: Mother Report and CCP at Schooi, signs were 
reversed on the mean difference, effect size, and improvement index to demonstrate that the treatment group was favored when negative differences were reported. The effect 
size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome 
that can be expected if that student is given the intervention. The improvement index is an aiternate presentation of the effect size, rejecting the change in an average student’s 
percentiie rank that can be expected if the student is given the intervention. PT = Parent Training; TT = Teacher Training; CT = Chiid Training 

® The means reported here were adjusted for pretest scores by the study authors. The p-vaiue ranges presented here were reported in the originai study (Reid et ai., 2003). A correc- 
tion for muitipie comparisons was needed and resuited in significance ieveis that differ from those in the originai study. Due to the muitipie comparisons adjustment, the p-vaiue for 
the contrast between the CT and comparison group on CCP at Home: Mother Report was higher than the criticai p-vaiue for statisticai significance; therefore, the WWC does not find 
the resuit to be statisticaiiy significant. The p-vaiue for CCP at Schooi was aiso higher than the criticai p-vaiue for statisticai significance for aii contrasts; therefore, the WWC does 
not find these resuits to be statisticaiiy significant. The originai study reported findings on three externai behavior outcomes; the father-reported outcomes are not presented in this 
appendix due to high attrition. The WWC muitipie comparison correction accounts for the fact that three measures were used in the articie to measure externai behavior. 
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Appendix D.2: Partial implementation findings for the social outcomes domain 



Outcome measure 


Study 

sample 


Sample 

size 


Mean 

(standard deviation) 


WWC calculations 




Intervention 

group 


Comparison 

group 


Mean 

difference 


Effect 

size 


Improvement 

index 


p-value 


Webster-Stratton et al., 2004^ 


Child Social Competence 


Ages 4-8 


56 


46.12 


44.13 


1.99 


nr 


na 


>0.05 


(CSC) with Peers 


(PT) 




(15.38) 


(15.41) 










Child Social Competence 


Ages 4-8 


49 


49.05 


44.13 


4.92 


nr 


na 


>0.05 


(CSC) with Peers 


(PT -H TT) 




(14.81) 


(15.41) 










Child Social Competence 


Ages 4-8 


56 


50.02 


44.13 


5.89 


0.35 


14 


<0.05 


(CSC) with Peers 


(CT) 




(14.85) 


(15.41) 










Child Social Competence 


Ages 4-8 


49 


49.55 


44.13 


5.42 


0.29 


11 


<0.10 


(CSC) with Peers 


(CT -H TT) 




(14.20) 


(15.41) 











Table Notes: This appendix presents comparisons between chiidren who received partiai versions of The Incredible Years (PT, CT, PT + TT, and CT + TT) and chiidren in the 
comparison group on measures that faii in the sociai outcomes domain. These are anciilary comparisons for the purposes of this review. Comparisons between chiidren who 
received the fuii version of The Incredible Tears and chiidren in the comparison group were used for rating purposes and are presented in Appendix C.2. Positive resuits for mean 
difference, effect size, and improvement index favor the intervention group; negative resuits favor the comparison group. The effect size is a standardized measure of the effect of 
an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can be expected if that student is given the 
intervention. The improvement index is an aiternate presentation of the effect size, refiecting the change in an average student’s percentiie rank that can be expected if the student 
is given the intervention. PT = Parent Training; TT = Teacher Training; CT = Chiid Training; nr = not reported; na = not appiicabie 

® The means reported here were adjusted for pretest scores by the study authors. The p-vaiue range presented here was reported in the originai study (Reid et ai., 2003). No correc- 
tions for ciustering or muitipie comparisons were needed. 
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Appendix El: Criteria used to determine the rating of a study 



1 study rating 


Criteria 


Meets evidence standards 


A Study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 


Meets evidence standards 
with reservations 


A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 
attrition that has established equivalence of the analytic samples. 


Appendix E2: Criteria used to determine the rating of effectiveness for an intervention 


1 Rating of effectiveness 


Criteria 


Positive effects 


Two or more studies show statistically significant positive effects, at least one of which meets WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 


Potentially positive effects 


At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 


Mixed effects 


At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 


Potentially negative effects 


One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 


Negative effects 


Two or more studies show statistically significant negative effects, at least one of which meets WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 


No discernible effects 


None of the studies shows a statistically significant or substantively important effect, either positive or negative. 


Appendix E3: Criteria used to determine the extent of evidence for an intervention 


1 Extent of evidence 


Criteria 


Medium to large 


The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 


Small 


The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Endnotes 

■' The descriptive information for this program was obtained from a publicly available source: the program’s website (http://www. 
incredibleyears.com/, downloaded January 2011). The WWC requests developers to review the program description sections for 
accuracy from their perspective. The program description was provided to the developer in January 201 1 . Further verification of the 
accuracy of the descriptive information for this program is beyond the scope of this review. The literature search reflects documents 
publicly available by August 201 1 . 

^ The studies In this report were reviewed using WWC Evidence Standards, Version 2.0, as described in protocol Version 2.0. The evi- 
dence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 
® For criteria used in the determination of the rating of effectiveness and extent of evidence, see Appendix E. These improvement 
index numbers show the average and range of student-level Improvement Indices for all findings across the studies. The one study 
that meets WWC evidence standards did not examine the effectiveness of The Incredible Years on children classified as having an 
emotional disturbance In the emotlonal/Internal behavior, reading achievement/literacy, math achievement, school attendance, or other 
academic performance domains. 

Cost information from Foster, E. M., Olchowski, A. E., & Webster-Stratton, C. (2007). Implementing behavioral Intervention components 
In a cost-effective manner: Analysis of The Incredible Years program. Journal of Early and Intensive Behavioral Intervention, 4(1 ), 284-304. 
® Webster-Stratton et al. (2004) reported separate comparisons between the wait-list comparison group and each of the treatment 
conditions. The comparison between children who received the full version of The Incredible Years (PT + TT + CT) and children in the 
comparison group is presented in Appendices C.1 and C.2 and forms the basis of the intervention rating; this comparison was chosen 
as the basis of the intervention rating because it evaluates the combined effects of The Incredible Years’ teacher, parent, and child 
trainings. All other comparisons are presented in Appendices D.1 and D.2 and do not contribute to the intervention rating. 

® Students in the treatment conditions were assessed again 1 and 2 years later. Webster-Stratton et al. (2004) and Reid, Webster-Stratton, 
and Hammond (2003) were not able to report comparisons between children In the treatment and comparison groups during the 1- and 
2-year follow-ups because the comparison group students received The Incredible Years program during the period between the original 
posttest and the follow-up measurements. Thus, outcomes from the 1 - and 2-year follow-ups are not included in this review. 

^ Webster-Stratton et al. (2004) also reported outcomes on negative and positive parenting, classroom management, classroom atmo- 
sphere, and parent/teacher satisfaction with the program. These outcomes are not presented in this report because they do not fall 
under a domain specified in the protocol. 

® The composite score used to measure fathers' reports on Child Conduct Problems (CCP) at home was Identical to the CCP at Home: 
Mother Report with one exception; the Eyberg Child Behavior Inventory (ECBI) Total Intensity score was completed by the father for 
the CCP at Home: Father Report. Mothers’ reports on CCP are presented in Appendix C.1 and form the basis of the intervention rat- 
ing. Comparisons based on the fathers’ composite score are not included in the intervention rating. Due to high levels of attrition on the 
fathers’ reports, comparisons between the comparison group and the PT + TT + CT, PT, and PT + TT groups meet evidence standards 
with reservations, and comparisons between the comparison group and the CT and CT + TT groups do not meet evidence standards. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse (2011, November). 
Children Classified as Having an Emotional Disturbance intervention report: The Incredible Years. Retrieved 
from http://whatworks.ed.gov. 
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Glossary of terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 
Standard deviation 
Statistical significance 
Substantively important 



Attrition occurs when an outcome variable is not avaiiable for ail participants Initially 
assigned to the intervention and comparison groups. The WWC considers the total attrition 
rate and the difference in attrition rates across groups within a study. 

If treatment assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a standardized measure of the magnitude of an effect that is comparable 
across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses a 
causal design (RCT or QED). 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent of 
evidence levels are given in Table E3. 

Along a percentile distribution of students, the improvement index represents the gain or 
loss of the average student due to the intervention. As the average student starts at the 
50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust the 
statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned to 
treatment and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into treatment and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality 
of the research design and the magnitude, statistical significance, and consistency 
in findings. The criteria for the ratings of effectiveness are given in Table E2. 

The standard deviation across all students in a group shows how dispersed the outcomes are. 
A measure with a small standard deviation would indicate that participants had more similar 
outcomes than a measure with a large standard deviation. 

Statistical significance is the probability that the difference between groups is a result of chance 
rather than a real difference between the groups. The WWC labels a finding statistically significant 
if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 



Please see the WWC Procedures and Standards Handbook (version 2.0) for additional details. 
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